5 research outputs found
Deep Connectomics Networks: Neural Network Architectures Inspired by Neuronal Networks
The interplay between inter-neuronal network topology and cognition has been
studied deeply by connectomics researchers and network scientists, which is
crucial towards understanding the remarkable efficacy of biological neural
networks. Curiously, the deep learning revolution that revived neural networks
has not paid much attention to topological aspects. The architectures of deep
neural networks (DNNs) do not resemble their biological counterparts in the
topological sense. We bridge this gap by presenting initial results of Deep
Connectomics Networks (DCNs) as DNNs with topologies inspired by real-world
neuronal networks. We show high classification accuracy obtained by DCNs whose
architecture was inspired by the biological neuronal networks of C. Elegans and
the mouse visual cortex.Comment: Presented at the Real Neurons & Hidden Units Workshop, 33rd
Conference on Neural Information ProcessingSystems (NeurIPS 2019), Vancouver,
Canad
Fonts-2-Handwriting: A Seed-Augment-Train framework for universal digit classification
In this paper, we propose a Seed-Augment-Train/Transfer (SAT) framework that
contains a synthetic seed image dataset generation procedure for languages with
different numeral systems using freely available open font file datasets. This
seed dataset of images is then augmented to create a purely synthetic training
dataset, which is in turn used to train a deep neural network and test on
held-out real world handwritten digits dataset spanning five Indic scripts,
Kannada, Tamil, Gujarati, Malayalam, and Devanagari. We showcase the efficacy
of this approach both qualitatively, by training a Boundary-seeking GAN (BGAN)
that generates realistic digit images in the five languages, and also
quantitatively by testing a CNN trained on the synthetic data on the real-world
datasets. This establishes not only an interesting nexus between the
font-datasets-world and transfer learning but also provides a recipe for
universal-digit classification in any script.Comment: Published as a workshop paper at ICLR 2019 (DeepGenStruct-2019
BERT Learns (and Teaches) Chemistry
Modern computational organic chemistry is becoming increasingly data-driven.
There remain a large number of important unsolved problems in this area such as
product prediction given reactants, drug discovery, and metric-optimized
molecule synthesis, but efforts to solve these problems using machine learning
have also increased in recent years. In this work, we propose the use of
attention to study functional groups and other property-impacting molecular
substructures from a data-driven perspective, using a transformer-based model
(BERT) on datasets of string representations of molecules and analyzing the
behavior of its attention heads. We then apply the representations of
functional groups and atoms learned by the model to tackle problems of
toxicity, solubility, drug-likeness, and synthesis accessibility on smaller
datasets using the learned representations as features for graph convolution
and attention models on the graph structure of molecules, as well as
fine-tuning of BERT. Finally, we propose the use of attention visualization as
a helpful tool for chemistry practitioners and students to quickly identify
important substructures in various chemical properties.Comment: 10 pages, 5 figure
Covering up bias in CelebA-like datasets with Markov blankets: A post-hoc cure for attribute prior avoidance
Attribute prior avoidance entails subconscious or willful non-modeling of
(meta)attributes that datasets are oft born with, such as the 40 semantic
facial attributes associated with the CelebA and CelebA-HQ datasets. The
consequences of this infirmity, we discover, are especially stark in
state-of-the-art deep generative models learned on these datasets that just
model the pixel-space measurements, resulting in an inter-attribute bias-laden
latent space. This viscerally manifests itself when we perform face
manipulation experiments based on latent vector interpolations. In this paper,
we address this and propose a post-hoc solution that utilizes an Ising
attribute prior learned in the attribute space and showcase its efficacy via
qualitative experiments.Comment: Accepted for presentation at the first workshop on Invertible Neural
Networks and Normalizing Flows (ICML 2019), Long Beach, CA, US
Understanding Adversarial Robustness Through Loss Landscape Geometries
The pursuit of explaining and improving generalization in deep learning has
elicited efforts both in regularization techniques as well as visualization
techniques of the loss surface geometry. The latter is related to the intuition
prevalent in the community that flatter local optima leads to lower
generalization error. In this paper, we harness the state-of-the-art "filter
normalization" technique of loss-surface visualization to qualitatively
understand the consequences of using adversarial training data augmentation as
the explicit regularization technique of choice. Much to our surprise, we
discover that this oft deployed adversarial augmentation technique does not
actually result in "flatter" loss-landscapes, which requires rethinking
adversarial training generalization, and the relationship between
generalization and loss landscapes geometries.Comment: Presented at the ICML 2019 Workshop on Uncertainty and Robustness in
Deep Learning, and CVPR 2019 Workshop on The Bright and Dark Sides of
Computer Vision: Challenges and Opportunities for Privacy and Security
(CV-COPS